[confgenerator] Add `file offset` storage to Otel Logging receivers. #2166

franciscovalentecastro · 2025-12-19T19:31:06Z

Description

Configuring the file_storage extension to be used as storage in Otel Logging receivers.

Some details :

Added to systemd, files and windows_event_log logging receiver.
The following folders will be used to store the bookmarks :
- Linux : /var/lib/google-cloud-ops-agent/opentelemetry-collector\file_storage/
- Windows : C:\ProgramData\Google\Cloud Operations\Ops Agent\run\file_storage

Related issue

b/469432672

How has this been tested?

Checklist:

Unit tests
- Unit tests do not apply.
- Unit tests have been added/modified and passed for this PR.
Integration tests
- Integration tests do not apply.
- Integration tests have been added/modified and passed for this PR.
Documentation
- This PR introduces no user visible changes.
- This PR introduces user visible changes and the corresponding documentation change has been made.
Minor version bump
- This PR introduces no new features.
- This PR introduces new features, and there is a separate PR to bump the minor version since the last release already.
- This PR bumps the version.

confgenerator/logging_receivers.go

systemd/google-cloud-ops-agent-opentelemetry-collector.service

jefferbrecht · 2026-01-12T20:20:54Z

Any thoughts on file_storage vs. db_storage? Do you have any concerns about unbounded retention of offsets for scenarios where someone configures a wildcard path?

confgenerator/logging_receivers.go

integration_test/ops_agent_test/main_test.go

cmd/ops_agent_windows/main_windows.go

jefferbrecht · 2026-02-02T18:27:13Z

integration_test/ops_agent_test/main_test.go

+		if len(matchingLogs) == 0 {
+			t.Errorf(`Expected to find at least one instance of "first line" log in the backend. Found %d instances.`, len(matchingLogs))
+		}
+		if len(matchingLogs) >= 2 {
+			t.Errorf(`Expected to find only one instance of "first line" log in the backend. Found %d instances.`, len(matchingLogs))
+		}


Nit: these can be combined into a single if len(matchingLogs) != 1 check

Just for completeness, this is the result of running locally the last iteration of this test (with the != 1 check) locally (without the file_storage update) :

=== NAME TestFileOffset/debian-cloud:debian-11/otel_logging agents.go:1029: Instance Log: [redacted] main_test.go:6008: Expected to find exactly one instance of "first line" log in the backend. Found 2 instances. --- FAIL: TestFileOffset (0.00s) --- FAIL: TestFileOffset/debian-cloud:debian-11 (0.00s) --- PASS: TestFileOffset/debian-cloud:debian-11/default (305.37s) --- FAIL: TestFileOffset/debian-cloud:debian-11/otel_logging (306.46s) FAIL

jefferbrecht · 2026-02-02T18:40:38Z

go.sum

 github.com/GoogleCloudPlatform/google-guest-agent v0.0.0-20250924181420-23412fbd6228/go.mod h1:31SvkAl6ORtir1odRpTl92XpTtn52GnBAHPtCKDSePo=
-github.com/GoogleCloudPlatform/opentelemetry-operations-collector/integration_test/gce-testing-internal v0.0.0-20260119145159-eff68fc7b8ef h1:v2hZ/3wkAB/9TvEqqE3zjlVsqozs4rzZ4zqvW/9YPLQ=
-github.com/GoogleCloudPlatform/opentelemetry-operations-collector/integration_test/gce-testing-internal v0.0.0-20260119145159-eff68fc7b8ef/go.mod h1:hmqK+hsg2z+Xo/Yz16sNxb6TZX+q0KxKsrgYzdpV0P8=
+github.com/GoogleCloudPlatform/opentelemetry-operations-collector/integration_test/gce-testing-internal v0.0.0-20260130171200-8bbc62976af0 h1:QeKYB2SRm5fOk6l91p8kNd6XkDBk/nwaeWnTwwBqwYQ=


Nit: run go mod tidy

jefferbrecht · 2026-02-02T18:42:09Z

LGTM modulo some remaining nits

… function.

Co-authored-by: Jeff Erbrecht <89024676+jefferbrecht@users.noreply.github.com>

franciscovalentecastro force-pushed the fcovalente-bookmarking branch from dfb9619 to 6bae68d Compare January 12, 2026 18:29

franciscovalentecastro requested a review from jefferbrecht January 12, 2026 18:31

jefferbrecht reviewed Jan 12, 2026

View reviewed changes

confgenerator/logging_receivers.go Show resolved Hide resolved

jefferbrecht reviewed Jan 12, 2026

View reviewed changes

systemd/google-cloud-ops-agent-opentelemetry-collector.service Show resolved Hide resolved

franciscovalentecastro commented Jan 12, 2026

View reviewed changes

confgenerator/logging_receivers.go Show resolved Hide resolved

franciscovalentecastro changed the title ~~[confgenerator] Add bookmarking to Otel Logging receivers.~~ [confgenerator] Add file offset storage to Otel Logging receivers. Jan 14, 2026

franciscovalentecastro force-pushed the fcovalente-bookmarking branch from a9b9bcd to 3805ae5 Compare January 14, 2026 23:44

jefferbrecht reviewed Jan 15, 2026

View reviewed changes

integration_test/ops_agent_test/main_test.go Outdated Show resolved Hide resolved

franciscovalentecastro mentioned this pull request Jan 19, 2026

[gce_testing] Create findMatchingLogs to return all found logs and create QueryAllLogs. GoogleCloudPlatform/opentelemetry-operations-collector#485

Merged

franciscovalentecastro force-pushed the fcovalente-bookmarking branch from 3bec296 to c0b1e2e Compare January 19, 2026 23:09

franciscovalentecastro force-pushed the fcovalente-bookmarking branch from af65ac7 to 606ed8c Compare January 29, 2026 18:08

franciscovalentecastro added the kokoro:force-run Forces kokoro to run integration tests on a CL label Jan 30, 2026

stackdriver-instrumentation-release removed the kokoro:force-run Forces kokoro to run integration tests on a CL label Jan 30, 2026

franciscovalentecastro added the kokoro:force-run Forces kokoro to run integration tests on a CL label Jan 30, 2026

stackdriver-instrumentation-release removed the kokoro:force-run Forces kokoro to run integration tests on a CL label Jan 30, 2026

franciscovalentecastro added the kokoro:force-run Forces kokoro to run integration tests on a CL label Jan 31, 2026

stackdriver-instrumentation-release removed the kokoro:force-run Forces kokoro to run integration tests on a CL label Jan 31, 2026

jefferbrecht reviewed Feb 2, 2026

View reviewed changes

cmd/ops_agent_windows/main_windows.go Outdated Show resolved Hide resolved

jefferbrecht reviewed Feb 2, 2026

View reviewed changes

jefferbrecht approved these changes Feb 2, 2026

View reviewed changes

franciscovalentecastro added 7 commits February 2, 2026 14:15

Add bookmarking (set storage) to all Otel Logging receivers.

1255aaa

Update goldens.

6ea8df8

Add new folder paths to docs.

4ea800e

Remove use of expOtlpExporter after rebase.

a419c22

Create TestLogCursor to verify log cursor is preserved after restart.

dfe8180

Rename test TestFileOffset and add comments.

6a5cd4a

Simplify TestFileOffset test using QueryAllLogs new "gce_testing"…

74934f1

… function.

franciscovalentecastro and others added 10 commits February 2, 2026 14:15

Update gce_testing commit.

2071b9f

Update workDirForImage with a non-root directory.

3d809bf

Update gce_testing commmit.

dcba6a3

Fix TestFileOffset after rebase.

7eca124

Update commit.

9a9a93c

Newest otelopscol commit.

5ee4d51

Update to otelopscol@master.

b523e79

Update cmd/ops_agent_windows/main_windows.go

466e7f4

Co-authored-by: Jeff Erbrecht <89024676+jefferbrecht@users.noreply.github.com>

Simplify test assertion for one log.

782e432

Run go mod tidy.

f5abf99

franciscovalentecastro force-pushed the fcovalente-bookmarking branch from c803629 to f5abf99 Compare February 2, 2026 19:16

franciscovalentecastro added 2 commits February 2, 2026 19:22

Update dev-docs comments.

08a3f6b

Reword test error log.

8ea55b5

franciscovalentecastro merged commit f462b7e into master Feb 2, 2026
74 of 97 checks passed

franciscovalentecastro deleted the fcovalente-bookmarking branch February 2, 2026 23:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[confgenerator] Add `file offset` storage to Otel Logging receivers. #2166

[confgenerator] Add `file offset` storage to Otel Logging receivers. #2166

franciscovalentecastro commented Dec 19, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

jefferbrecht commented Jan 12, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jefferbrecht Feb 2, 2026

Uh oh!

franciscovalentecastro Feb 2, 2026

Uh oh!

franciscovalentecastro Feb 2, 2026

Uh oh!

jefferbrecht Feb 2, 2026

Uh oh!

franciscovalentecastro Feb 2, 2026

Uh oh!

jefferbrecht commented Feb 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[confgenerator] Add file offset storage to Otel Logging receivers. #2166

[confgenerator] Add file offset storage to Otel Logging receivers. #2166

Conversation

franciscovalentecastro commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related issue

How has this been tested?

Checklist:

Uh oh!

Uh oh!

Uh oh!

jefferbrecht commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jefferbrecht Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

franciscovalentecastro Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

franciscovalentecastro Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

jefferbrecht Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

franciscovalentecastro Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

jefferbrecht commented Feb 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[confgenerator] Add `file offset` storage to Otel Logging receivers. #2166

[confgenerator] Add `file offset` storage to Otel Logging receivers. #2166

franciscovalentecastro commented Dec 19, 2025 •

edited

Loading

jefferbrecht commented Jan 12, 2026 •

edited

Loading